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ABSTRACT 


Nowadays an educational mobile application has been widely accepted 
and opened new windows of opportunity to explore. With its flexibility 
and practicality, the mobile application can promote learning through playing 
with an interactive environment especially to the children. This paper 
describes the development of mobile learning to help children above 4 years 
old in learning English and Arabic language in a playful and fun way. 
The application is developed with a combination of Android Studio 
and the machine learning technique, TensorFlow object detection API in 
order to predict the output result. Developed application namely 
“LearnWithIman” has successfully been implemented and the results show 
the prediction of application is accurate based on the captured image with 
the list item. The inclusion of the user database for lesson tracking and new 


Object detection API lesson will be added for improvement in the future. 
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1. INTRODUCTION 

Mobile devices such as smart phones and tablets are becoming part of human’s daily life 
communication, as well as entertainment. With the advanced of mobile technologies, it can provide 
an interactive learning content application for children educational purpose which can be simply accessed 
anywhere and anytime. The fact that children are best learns through playing, the making of an interactive 
application for Android mobile attracted the research attention. There is a high potential in the usage 
of mobile technology, in term of how a language can be learned and enhanced. This might be a big help 
for children to become proficient in second language at early age. Stockwell in his paper shows that 
the effects of vocabulary learning through mobile platforms are comparable with the traditional ways [1]. 
This is also supported by Basal et al. through the experiment done that proved the effectiveness of learning 
vocabulary through mobile applications compared through the traditional paper-based activities [2]. 
Vocabulary learning is very crucial to build the blocks of language such as read comprehension 
and language speaking [3]. 
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Numerous studies have reported on the use of mobile devices to develop language skills through 
language learning application in past years. Makoe and Shandu implemented a mobile-based application 
to develop vocabulary learning mobile app to enhance the English vocabulary teaching and learning [4]. 
Wang also develop a mobile app to improve the college student’s English vocabulary learning with both 
English and Chinese description [5]. While Sweeney and Moore are investigate the potential of mobile 
devices to be used in language learning. They found that language learning through mobiles has high 
potential but with the collaboration of both developer and educatorto make sure that it can benefit and help 
the students [6]. The most common method for mobile application development using image processing 
machine learning technique is Mobile Cloud Computing (MCC). MCC is combination of mobiles and cloud 
computing which can simply processing from the user local devices to the data centre facilities over 
the internet [7-24]. The major service providers of MCC are including Google, Amazon, Microsoft 
and Yahoo. Figure 1 shows the general architecture of MCC. 





Mobile User Internet Data centre - 
cloud storage 


Figure 1. Mobile cloud computing (MCC) architecture 


The users can simply accessing the cloud storage as long as the internet connection is there. 
However, this can be disadvantage of the MCC. The overall performance of the machine learning can 
be affected depended on the imternet connection speed which it cannot perform inference locally 
on the device without internet [25-26]. Thus, TensorFlow technique can be a solution to the running machine 
learning on mobile devices. TensorFlow is able to runs the machine learning inference computation locally 
on the device. With low latency and small binary size, TensorFlow also can be a lot faster and more reliable 
than MCC [27-28]. This paper presents the development of Android mobile application namely 
“LeamWithIman” for language learning through capturing image based on the word given, that aim 
for toddlers above 4 years old to enhance their English and Arabic vocabulary. TensorFLow deep learning 
techniques will be used to train the models of captured image for image prediction. 


2. METHOD 
2.1. Android application development 

In order to attract the children interest, “Learn WithIman” is developed to be a fun application with 
the learning through playing concept. “LearnWithIman” introduces two basic languages of English 
and Arabic which divided by different categories. Prior to the application development, the selection of each 
category is crucial to match the level of the children age. The categories are including the ‘Read Easy’, 
‘School Bag’, ‘Sweet Home’, ‘Fruits and Veggies’, ‘Speak Arabic’ and ‘Explore Science’. 

For the development of the Android mobile application namely “LearnWithIman”, the TensorFlow 
deep learning technique and Android Studio were used. As mention in Introduction Section, TensorFlow will 
be used as a machine learning method to predict the image captured from “LearmWithIman” application. 
“Learn WithIman” application is preloaded with TensorFlow object detection API which uses single shot 
detector (SSD) with MobileNet pre-trained model that is trained on Microsoft Common Object in Context 
(COCO) dataset. The captured tmage will be compared with the preloaded image dataset with the highest 
score in order to predict the output result. Android Studio is used to customize the look of the application 
in Android platform. Figure 2 provides the architecture of “LearnWithIman” android mobile application. 

Upon entering “LeamWithIman” application, the user is presented with its main interface 
for categories selection as shown in Figure 3. The application will then display the name of the item with 
sound that requires the user to capture the image of the item. TensorFlow will perform the prediction 
by giving the score of the captured image. Once the click button is pressed, the application will compare 
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the score with the name of the item. The correct sign will be displayed if the captured image is matched with 
the name of the item. This will be repeated until all the items are displayed. The point will be shown 
at the end of application. 
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Figure 2. “Learn WithIman” application architecture 








Figure 3. The main interface of “Learn WithIman” application 


The Flowchart of “learnWithIman’ application is shown in Figure 4. While Figure 5 presents 
the screen of image capturing from the application. There are three buttons are available as illustrated in 
Figure 5, left button is the count of correct item based on the captured image, middle button is to capture 
the image from the rear camera and the nght button is to skip the current item and go to the next item. 
The name of the item is displayed at the bottom of the application. 
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Figure 4. The flowchart of “Learn WithIman” android application 


Capture the image 








Point Next item 


Figure 5. Screen of application to capture the image 


The name of the item 
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3. RESULTS AND DISCUSSION 

Several experiments have been done with different requirements to observe the practicability 
of the application in image detection and prediction. At first, the application was tested with random 
categories as shown in Table 1. Then, the test for the same item with the different captured image 
was Observed to check the reliability of the application. Last but not least, to see the effect of the image 
background forthe application prediction. Table 1 presents the comparing scores between the captured image 
and the list item through TensorFlow Object Detection API. As can be seen from the table, “LeamWithIman” 
is able to detect the captured image accurately according to the list item. On average, the detection time 
of the captured image took only 100 ms which considered as sufficient enough. 


Table 1. The score of the matched item from different categories 
Category-Name of the Item Captured image Detection time (ms) Score 








Read easy-mug 105 0.58 
School bag-sharpener 136 0.95 
sharpener 
a 
0.02 
Sweet home-pillow 74 0.86 


pillow 


Table 2 illustrates the prediction of the same item but different captured image. It is apparent from 
the table that “LeamWithIman” consistently matches the item correctly even though the image captured 
is different. The results shown indicate that the prediction of the application is reliable. Table 3 provides 
the results of the same item and captured image but with different background. The results obtained revealed 
that “LearnWithIman” able to detect the matched image even with a different background. 
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Table 2. The result from same item with different image 
Capturedimage Score Prediction 


0.71 Matched 
0.84 Matched 
0.87 Matched 





Table 3. The result of same image with different background 
Captured Image Score Prediction 


0.83 Matched 
0.69 Matched 
0.55 Matched 
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4. CONCLUSION 

This work has presented the development of Android application for children in language learning 
namely “LearnWithIman”. The developed system is able to perform the prediction of the image captured 
by “LeamWithIman” application for children language learning with high accuracy. The application 
is implemented with a combination of interactive elements, visual and sound for pronunciation to attract 
the children interest to learn through mobile application. As for future work, the inclusion of the user 


database for lesson tracking will be considered. Moreover, new lessons for language learning will 
be added as well. 
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